Compact Fluorescence Endoscopy Video System

ABSTRACT

A fluorescence endoscopy video system includes a multimode light source for producing white light, fluorescence excitation light, or fluorescence excitation light with a reference reflectance light. An endoscope directs light to illuminate a tissue sample and collects reflected light or fluorescence light produced by the tissue. A camera includes a high sensitivity color image sensor having a plurality of pixel elements. Each of the pixel elements has an integrated filter configured to block reflected excitation light from reaching the pixel elements and allow fluorescence and reflectance light to reach the pixel elements. A processor receives image signals from the image sensor, combines image signals from a first group of pixel elements to form a first image formed by fluorescence light, and combines image signals from a second group of pixel elements to form a second image formed by reflectance light. A video monitor simultaneously superimposes the first and second images.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/761,462, filed Apr. 16, 2010, pending, which is a divisional of U.S.patent application Ser. No. 11/969,974, filed Jan. 7, 2008, now U.S.Pat. No. 7,722,534, which is a divisional of U.S. patent applicationSer. No. 10/899,648, filed Jul. 26, 2004, now U.S. Pat. No. 7,341,557,which is a divisional of U.S. patent application Ser. No. 09/905,642,filed Jul. 13, 2001, now U.S. Pat. No. 6,821,245, which is acontinuation-in-part of U.S. patent application Ser. No. 09/615,965,filed Jul. 14, 2000, now abandoned, the benefit of the filing date ofwhich is being claimed under 35 U.S.C. §120.

TECHNICAL FIELD

The present disclosure relates to medical imaging systems in generaland, in particular, to fluorescence endoscopy video systems.

BACKGROUND

Fluorescence endoscopy utilizes differences in the fluorescence responseof normal tissue and tissue suspicious for early cancer as a tool in thedetection and localization of such cancer. The fluorescing compounds orfluorophores that are excited during fluorescence endoscopy may beexogenously applied photoactive drugs that accumulate preferentially insuspicious tissues, or they may be the endogenous fluorophores that arepresent in all tissue. In the latter case, the fluorescence from thetissue is typically referred to as autofluorescence or nativefluorescence. Tissue autofluorescence is typically due to fluorophoreswith absorption bands in the UV and blue portions of the visiblespectrum and emission bands in the green to red portions of the visiblespectrum. In tissue suspicious for early cancer, the green portion ofthe autofluorescence spectrum is significantly suppressed. Fluorescenceendoscopy that is based on tissue autofluorescence utilizes thisspectral difference to distinguish normal from suspicious tissue.

Since the concentration and/or quantum efficiency of the endogenousfluorophores in tissue is relatively low, the fluorescence emitted bythese fluorophores is not typically visible to the naked eye.Fluorescence endoscopy is consequently performed by employing low lightimage sensors to acquire images of the fluorescing tissue through theendoscope. The images acquired by these sensors are most often encodedas video signals and displayed on a color video monitor. Representativefluorescence endoscopy video systems that image tissue autofluorescenceare disclosed in U.S. Pat. No. 5,507,287, issued to Palcic et al.; No.5,590,660, issued to MacAulay et al.; No. 5,827,190, issued to Palcic etal.; and No. 5,647,368, issued to Zeng et al. Each of these patents isassigned to Xillix Technologies Corp. of Richmond, British Columbia,Canada, the assignee of the present application. While the systemsdisclosed in the above-referenced patents are significant advances inthe field of early cancer detection, improvements can be made.

These aforementioned systems are typically used in conjunction with anendoscope to which a camera containing low light sensors is attached orutilize a video endoscope with the camera located at the insertion endof the endoscope. In particular, it is desirable to reduce the size,cost, and weight of the camera described for these systems. Sincefluorescence endoscopy is commonly performed as an adjunct toconventional white light endoscopy, it is also desirable for the systemto be capable of acquiring both color and fluorescence images with thesame camera and light source. It is also desirable to optimize such afluorescence endoscopy video system to detect various types of cancer indifferent organs and to provide features so that it is easily calibratedfor use with different types of endoscopes. It is also desirable thatsuch a system be compatible for use with exogenously applied photoactivedrugs. Finally, there is a need for a system in which the contrastbetween normal and suspicious tissue may be enhanced in the displayedfluorescence images.

SUMMARY OF THE INVENTION

A fluorescence endoscopy video system in accordance with the presentinvention includes:

an endoscopic light source that is capable of operating in multiplemodes to produce either white light, fluorescence excitation light, orfluorescence excitation light with a reference reflectance light;

an endoscope including a light guide for transmitting light to thetissue under observation and either an imaging guide or compact camerafor receiving light from the tissue under observation;

a compact camera that receives light from the image guide of anendoscope or directly from the tissue by virtue of being located in theinsertion portion of the endoscope and is capable of operating inmultiple imaging modes to acquire color or multichannel fluorescence andreflectance images. Images obtained are optically divided and projectedonto one or more image sensors by a fixed beam splitter in the camera.One of the beams from the beam splitter is directed to an image sensorthat acquires color images. The remaining beam is (or beams are) usedalone or in conjunction with the first beam to acquire fluorescenceand/or reflectance images;

an image processor and system controller digitize, process, and encodethe image signals as a color video signal;

a contrast enhancement function may be present in theprocessor/controller. This function applies a non-unity gain factor tothe processed reference image signal based on the relative intensity ofthe fluorescence/reflectance (or fluorescence/fluorescence) imagesignals;

a color video monitor displays the processed video images; and a colorcalibration mechanism allows the response of the system to be calibratedfor optical characteristics of different endoscopes and/or other imagesignal path variables.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same become betterunderstood by reference to the following detailed description, whentaken in conjunction with the accompanying drawings, wherein:

FIGS. 1A-1B are block diagrams of fluorescence endoscopy video systemsaccording to embodiments of the present invention;

FIG. 2 is a block diagram of a multimode light source in accordance withanother aspect of the present invention;

FIGS. 3A-3C illustrate a preferred embodiment of the camera that canacquire color; fluorescence/reflectance, and/orfluorescence/fluorescence images according to the present invention withoptional placement for collimation and imaging optics;

FIGS. 4A-4E illustrate a number of camera beam splitter configurations;

FIG. 5 illustrates a second embodiment of a camera according to thepresent invention;

FIG. 6 illustrates a third embodiment of a camera according to thepresent invention;

FIGS. 7A-7D illustrate a number of spectral splitter and filter assemblyconfigurations;

FIG. 8 illustrates a fourth embodiment of a camera according to thepresent invention;

FIG. 9A-9B illustrates examples of spectral splitter and filteringassembly that can transmit images to the same image plane;

FIG. 10 illustrates a fifth embodiment of a camera according to thepresent invention;

FIGS. 11A-11D are graphs illustrating presently preferred transmissioncharacteristics of filters and dichroic splitters forfluorescence/reflectance imaging using green fluorescence light and redreflectance light;

FIGS. 12A-12D are graphs illustrating presently preferred transmissioncharacteristics for filters and dichroic splitters forfluorescence/reflectance imaging using green fluorescence light and bluereflectance light;

FIGS. 13A-13D are graphs illustrating presently preferred transmissioncharacteristics of filters and dichroic splitters forfluorescence/reflectance imaging using red fluorescence light and bluereflectance light;

FIGS. 14A-14D are graphs illustrating presently preferred transmissioncharacteristics of filters and dichroic splitters forfluorescence/reflectance imaging using red fluorescence light and bluereflectance light;

FIGS. 15A-15D are graphs illustrating presently preferred transmissioncharacteristics of filters and dichroic splitters forfluorescence/reflectance imaging using red fluorescence light andnear-infrared reflectance light;

FIGS. 16A-16D are graphs illustrating presently preferred transmissioncharacteristics of filters and dichroic splitters forfluorescence/reflectance imaging using green fluorescence light andnear-infrared reflectance light;

FIGS. 17A-17D are graphs showing presently preferred transmissioncharacteristics of filters and dichroic splitters for use withfluorescence/fluorescence imaging;

FIG. 18 is a graph illustrating characteristics of a blue blockingfilter presently preferred transmission for fluorescence/reflectance orfluorescence/fluorescence imaging using a color image sensor withintegrated selective filters.

FIG. 19 is a block diagram of a system to perform color calibration ofthe fluorescence endoscopy video system according to another aspect ofthe present invention; and

FIGS. 20-22 are graphs showing contrast enhancement tests and functionsthat can be used to highlight potentially cancerous tissue in accordancewith another aspect of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1A is a block diagram of a fluorescence endoscopy video system 50in accordance with a presently preferred embodiment of the presentinvention. The system includes a multimode light source 52 thatgenerates a white light for obtaining color images. In a second mode ofoperation, the light source 52 produces an excitation light for inducingtissue autofluorescence. In a third mode of operation, the light source52 produces an excitation light for inducing tissue autofluorescence anda reference reflectance light. The use of excitation light andexcitation plus reflectance light for fluorescence/fluorescence andfluorescence/reflectance imaging modes will be described in furtherdetail below. Light from the light source 52 is supplied to anillumination guide 54 of an endoscope 60, which then illuminates atissue sample 58 that is to be imaged.

FIG. 2 shows the components of the light source 52 in greater detail.The light source 52 includes an arc lamp 70 that is surrounded by areflector 72. In the preferred embodiment of the invention, the arc lamp70 is a high pressure mercury arc lamp (such as the Osram VIP R 120P24).Alternatively, other arc lamps or broadband light sources may be used,but a high pressure mercury lamp is currently preferred for itscombination of high blue light output, reasonably flat white lightspectrum, and small arc size.

The light from the arc lamp 70 is coupled to a light guide 54 of anendoscope 60 through appropriate optics 74, 76, and 78 for lightcollection, spectral filtering and focusing respectively. The light fromthe arc lamp is spectrally filtered by one of a number of opticalfilters 76A, 76B, 76C . . . that operate to pass or reject desiredwavelengths of light in accordance with the operating mode of thesystem. For color imaging, optical filter 76A eliminates any spectralpeaks and modifies the color temperature of the light produced by thearc lamp 70. The transmission characteristics of the light sourcefilters 76B, 76C . . . for fluorescence/reflectance andfluorescence/fluorescence imaging modes, respectively, are discussed inconjunction with the characteristics of the camera filters 118, 119A,119B . . . below.

An intensity control 80 that adjusts the amount of light transmittedalong the light path is positioned at an appropriate location betweenthe arc lamp 70 and the endoscope light guide 54 and controls the amountof light coupled to the light guide 54. In addition, a shutter mechanism82 may be positioned in the same optical path in order to block any ofthe light from the lamp from reaching the light guide. A controller 86operates an actuator that moves the filters 76A, 76B, or 76C into andout of the light path. The controller 86 also controls the position ofthe intensity control 80 and the operation of the shutter mechanism 82.

As shown in FIG. 1A, the system also includes a multimode camera 100.The light that is collected from the tissue by the endoscope 60 istransmitted through an image guide 56 and is projected into themultimode camera 100. Because fluorescence endoscopy is generally usedas an adjunct to white light endoscopy, each of the various embodimentsof the camera described below may be used both for color andfluorescence/reflectance and/or fluorescence/fluorescence imaging.

FIG. 1B is a block diagram of an alternative fluorescence endoscopyvideo system 50, which differs from that shown in FIG. 1A, in that themultimode camera 100 is located at the insertion end of the endoscopeand the endoscope does not contain image guide 56. With thesedifferences, the resulting endoscope 60 can be characterized as afluorescence video endoscope, similar to video endoscopes currently onthe market (such as the Olympus CF-240L) in utility, but with theadditional ability to be utilized for both color andfluorescence/reflectance and/or fluorescence/fluorescence imaging.

Other than the location of the multimode camera 100 at the insertion endof the endoscope and the lack of an endoscope image guide 56, the systemof FIG. 1B is identical to that shown in FIG. 1A. The variousembodiments of the camera described below lend themselves toimplementation in a fluorescence video endoscope due to theircompactness.

In this alternative system, the multimode camera 100 directly collectsthe light emitted by the tissue. By locating the camera at the insertionend of the endoscope, the inherent advantages of a video endoscope canbe obtained: namely, the light available to form an image and the imageresolution are improved compared to the case when the image istransmitted outside the body through an endoscope imaging guide.

In the first embodiment, shown in FIG. 3A, a camera 100A receives lightfrom the image guide 56 of an endoscope 60 and directs the light towardsa color image sensor 102 and a low light image sensor 104. In prior artcamera designs, light is typically directed to either of the two imagesensors 102 or 104 with a movable mirror that is selectively insertedinto the optical path. Such a mirror must be carefully constructed sothat it moves within tight tolerances. This adds greatly to thecomplexity and cost of the camera. The need to maintain these tighttolerances throughout the lifetime of the system also decreases thecamera's reliability.

The camera 100A, according to the preferred embodiment of the presentinvention, replaces the moving mirror with a fixed optical beam splitter106 that splits the incoming light into two beams. The light beam issplit such that a smaller proportion of the light received from theendoscope 60 is directed towards the color image sensor 102 and a largerproportion of the incoming light is directed towards the low light imagesensor 104. In this embodiment, the beam splitter may be a standardcommercially available single plate 88, single cube 89, or singlepellicle design 90, as shown in FIGS. 4A-4C. It should be noted that, ifthe optical path between the endoscope 60 and image sensors contains anuneven number of reflections (e.g., such as from a single component beamsplitter), the image projected onto the sensor will be left-to-rightinverted. The orientation of such images will need to be corrected byimage processing.

In some instances, it is desirable that the light split by the splitter106 be projected in the same image plane. Therefore, the optical beamsplitter 106 may be a combination of simple components or a custom prismdesign as shown in FIGS. 4D-4E. The cube assembly shown in FIG. 4D is anexample of standard, commercially available glass components (beamsplitter cube 89, right angle prism 91, and simple glass block 92) thathave been combined into an assembly. Because the path of the light thatpasses through the right angle prism 91 is longer than that which passesthrough the beam splitter cube 89, for cases in which noncollimatedlight is being split by the splitter 106, the glass block 92 ispositioned behind the right angle prism 91 to compensate for thedifferent path lengths such that both beams are focused in the sameimage plane.

The custom prism shown in FIG. 4E is comprised of three prisms. A firstpartially-mirrored surface 95 on a first prism directs a portion of theincoming light toward a fully reflective surface 96 on the first prism.Light reflected off the surface 96 passes through a second prism 99.Light passing through the partially-mirrored surface 95 is reflected offfully reflective surfaces 97 and 98 of a third prism. The optical pathlength of the beam that is reflected by the partially mirrored surface95 is the same as the optical path of the light that passes through thepartially-mirrored surface 95.

The custom prism shown in FIG. 4E has the advantage that it is morecompact than the cube assembly and that it provides a continuous surfacefrom which the image sensor(s) may be located. In both of these versionsof the beam splitter, the two paths for the split image contain an evennumber of reflections and are optically equivalent in length. In thecase of an optical imaging configuration as described in FIG. 3C below,this allows both images to be projected into the same image plane (e.g.,such as would be required if both images were imaged with a single imagesensor).

In FIG. 3A, light collimating optics 110 are positioned between theendoscope 60 and beam splitter 106, and imaging optics 112 and 114 arepositioned immediately preceding the color image sensor 102 and the lowlight image sensor 104, respectively. In an alternative opticalconfiguration shown in FIG. 3B, the collimating optics 110 have beeneliminated. Such a configuration is preferable to that in FIG. 3A, ifthe light beam from the endoscope 60 is already collimated.

The presently preferred configuration of the camera 100A is shown inFIG. 3C. In this embodiment, the collimating optics 110 have beeneliminated and replaced with a single set of imaging optics 113 locatedbetween the endoscope 60 and beam splitter 106. The advantage of thisconfiguration is that all imaging is performed and controlled by thesame imaging optics 113. Such a configuration requires all beam paths tohave the same optical path length, however, and this restriction must beconsidered in the design of the beam splitter 106 and a pair of spectralfilters 118 and 119 that are located in the path to the image sensors102 and 104. Glass block 121 is inserted into the optical path whenspectral filter 119 is removed. In addition, the fact that these opticalelements are located in a converging beam path must be considered inspecifying these elements and in the design of the imaging optics 113.All of the options for the collimating and imaging optics describedabove, and their attendant benefits and drawbacks, also apply to thesubsequent descriptions of camera embodiments represented in FIGS. 5, 6,8, and 10.

As shown in FIGS. 3A-3C, a spectral filter 118 is located in the opticalpath between the beam splitter 106 and the low light image sensor 104.Alternatively, the spectral filter 118 may be incorporated as an elementof the beam splitter 106.

A second spectral filter 119 is positioned so that it can be moved intoand out of the optical path between the beam splitter 106 and the colorimage sensor 102. For the case in which beam splitting is occurring in anoncollimated beam path, when filter 119 is moved out of position, aglass block 121 with the same optical path length as filter 119 is movedinto position between the beam splitter 106 and the color image sensor102 to maintain a constant optical path length. Alternatively, thisinsertable spectral filter 119 and glass block 121 (if required) may beincorporated elsewhere in the optical path between the endoscope 60 andthe color image sensor 102. Moving a filter into and out of an opticalpath can be done with a simple mechanism as there are no stringentmechanical and optical requirements like those for moving a mirror.

The low light image sensor 104 preferably comprises a monochrome chargecoupled device (CCD), monochrome charge coupled device with chargecarrier multiplication (such as the Texas Instruments TC253 or theMarconi Technologies CCD65), intensified charge coupled device (ICCD),charge injection device (CID), charge modulation device (CMD),complementary metal oxide semiconductor image sensor (CMOS) or electronbeam charge coupled device (EBCCD) type of sensor. The color imagesensor 102 is preferably a color CCD, a color CCD with charge carriermultiplication, a three-CCD color image sensor assembly with chargecarrier multiplication, a three-CCD color image sensor assembly, a colorCMOS image sensor, or a three-CMOS color image sensor assembly.

As shown in FIG. 1A, the system also includes a processor/controller 64and a video monitor 66. The processor/controller 64 receives thetransduced image signals from the camera 100 and digitizes and processesthese signals. The processing of these signals may include theapplication of certain contrast enhancement algorithms described below.The processed signals are then encoded in a video format and displayedon a color video monitor 66.

Based on operator input, the processor/controller 64 also providescontrol functions for the fluorescence endoscopy video system. Thesecontrol functions include providing control signals that

-   -   control the camera gain in all imaging modes;    -   coordinate the imaging modes of the camera and light source;    -   provide a light level control signal for the light source, and    -   provide control signals for any image data management systems        that may be used to record and archive image data.

The reason that two separate images in different wavebands are acquiredin the fluorescence/reflectance and fluorescence/fluorescence modes offluorescence endoscopy video systems described herein will now beexplained. It is known that the intensity of the autofluorescence incertain wavebands changes as tissues become increasingly abnormal (i.e.,as they progress to frank cancer). When acquiring images within such awaveband of autofluorescence, however, it is not easy to distinguishbetween those changes in the signal strength that are due to pathologyand those that are due to imaging geometry and shadows. A secondfluorescence image or a reflected light image, acquired in a waveband inwhich the image signal is not significantly affected by tissuepathology, may be utilized as a reference signal with which the signalstrength of the first fluorescence image can be “normalized.”

This normalization may be performed by assigning each of the two imagesignals a different display color, e.g., by supplying the image signalsto different color inputs of a color video monitor. When displayed on acolor video monitor, the two images are effectively combined to form asingle image, the combined color of which represents the relativestrengths of the signals from the two images. Since the color of thecombined image is independent of the absolute strength of the separateimage signals, the color will not change as a result of changes in thedistance or angle of the endoscope 60 to the tissue sample 58 or otherimaging geometry factors. If, however, there is a change in the shape ofthe autofluorescence spectrum of the observed tissue that gives rise toa change in the relative strength of the two image signals, such achange will be represented as a change in the color of the displayedimage.

The mixture of colors with which normal tissue and tissue suspicious forearly cancer are displayed depends on the gain applied to each of thetwo separate image signals. There is an optimal gain ratio for whichtissue suspicious for early cancer in a fluorescence image will appearas a distinctly different color than normal tissue. This gain ratio issaid to provide the operator with the best combination of sensitivity(ability to detect suspect tissue) and specificity (ability todiscriminate correctly). If the gain applied to the reference imagesignal is too high compared to the gain applied to the fluorescenceimage signal, the number of tissue areas that appears suspicious butwhose pathology turns out to be normal, increases. Conversely, if therelative gain applied to the reference image signal is too low,sensitivity decreases and suspect tissue will appear like normal tissue.For optimal system performance, therefore, the ratio of the gainsapplied to the image signals must be maintained at all times.

In vivo spectroscopy has been used to determine which differences intissue autofluorescence and reflectance spectra have a pathologicalbasis. The properties of these spectra determine the particularwavebands of autofluorescence and reflected light required for thefluorescence/reflectance imaging mode, or the particular two wavebandsof autofluorescence required for fluorescence/fluorescence imaging mode.Since the properties of the spectra depend on the tissue type, thewavelengths of the important autofluorescence band(s) may depend on thetissue being imaged and the location within those tissues. Thespecifications of the optical filters described below are a consequenceof these spectral characteristics, and are chosen to be optimal for thetissues to be imaged.

The operation of the preferred embodiment of the fluorescence endoscopyvideo system will now be described. The camera 100 shown in FIG. 1 iscapable of color, fluorescence/reflectance, andfluorescence/fluorescence imaging modes. In the color imaging mode, theprocessor/controller 64 provides a control signal to the multimode lightsource 52 that it should be in white light mode. The light source 52selects and positions the appropriate optical filter 76A into theoptical path between the arc lamp 70 and the endoscope light guide 54.This filter 76A removes any spectral peaks and adjusts the colortemperature of the light produced by the arc lamp 70. The filtered lightfrom the light source 52 is projected into the endoscope light guide 54and is transmitted to the tip of the endoscope 60 to illuminate thetissue 58.

The processor/controller 64 also ensures that the camera is in thecorrect imaging mode to avoid damage to the sensitive low light imagesensor 104. In the case where the low light image sensor 104 is an ICCD,for example, the voltage across the photocathode is set to zero. Thelight reflected by the tissue 58 is collected by the endoscope imageguide 56 and is projected through the camera beam splitter 106 onto thecolor image sensor 102. Spectral filter 119 is removed from the opticalpath during this imaging mode and replaced by glass block 121 (ifrequired). The color image is transduced by the color image sensor 102and the resulting image signal is transmitted to theprocessor/controller 64.

Based on the brightness of the color image, the processor/controller 64provides a control signal to the multimode light source 52 to adjust theintensity control 80 and thereby adjust the level of light output by theendoscope 60. The processor/controller 64 may also send a control signalto the camera 100 to adjust the gain of the color image sensor 102.After being processed, the color image is displayed on the video monitor66. All of the imaging operations occur in real-time, that is to saythey occur at analog video display rates (30 frames-per-second for NTSCformat, and 25 frames-per-second for PAL format).

When switching to the fluorescence/reflectance imaging mode, theprocessor/controller 64 provides a control signal to the multimode lightsource 52 to indicate that it should be in fluorescence/reflectancemode. The light source 52 selects and positions the appropriate opticalfilter 76B into the optical path between the arc lamp 70 and theendoscope light guide 54. This filter 76B transmits those wavelengths oflight that will induce the tissue 58 under examination to fluoresce. Italso transmits reference reflectance light in either the green or redportions of the visible spectrum or alternatively, the blue excitationlight can be utilized for the reference. All other wavelengths of lightare blocked as described below. The filtered light is then projectedinto the endoscope light guide 54 and is transmitted to the tip of theendoscope 60 to illuminate the tissue 58.

The processor/controller 64 also ensures that the camera 100 is in thecorrect imaging mode by providing power to the low light image sensor104. The fluorescence emitted and reference light reflected by thetissue 58, along with the reflected excitation light, are collected bythe endoscope image guide 56 and are projected through the camera beamsplitter 106 onto the low light image sensor 104 and the color imagesensor 102. Spectral filter 118 limits the light transmitted to the lowlight image sensor 104 to either green or red autofluorescence lightonly and blocks the light in the excitation and reference wavebandstransmitted by light source filter 76B. Spectral filter 119 is insertedinto the optical path of the color image sensor 102 during this imagingmode and transmits only the reflected reference waveband light. Thereflectance light transmission specifications of filter 119 and lightsource filter 76B are chosen such that the intensity of the reflectedlight at the color image sensor 102 results in a transduced image signalwith good signal-to-noise characteristics and without significantsaturation, while at the same time allowing for excitation of sufficientautofluorescence for imaging. (Note that if spectral filter 119 waslocated between the beam splitter 106 and the endoscope 60, it wouldalso have to transmit the autofluorescence light detected by the lowlight image sensor 104.) The autofluorescence image is then transducedby the low light image sensor 104, the reference image is transduced bythe color image sensor 102, and the resulting image signals aretransmitted to the processor/controller 64.

Based on the brightness of the transduced images, theprocessor/controller 64 may provide a control signal to the multimodelight source 52 to adjust the intensity control 80 and thereby adjustthe level of light delivered to the endoscope 60. Theprocessor/controller 64 may also send control signals to the camera 100to adjust the gains of the low light image sensor 104 and the colorimage sensor 102 in order to maintain constant image brightness whilekeeping constant relative gain, as described in more detail below. Afterbeing processed, the images from the two sensors are combined into asingle image, which is displayed on the video monitor 66. Again, all ofthe imaging operations occur in real-time.

In order for the combined image to have optimal clinical meaning, for agiven proportion of fluorescence to reference light signals emitted bythe tissue and received by the system, it is necessary for a consistentproportion to also exist between the processed image signals that aredisplayed on the video monitor. This implies that the (light) signalresponse of the fluorescence endoscopy video system is calibrated.

Calibration of the signal response may be implemented in theprocessor/controller 64. To calibrate the system, the gain response ofthe fluorescence image sensor and reference image sensor arecharacterized, and those response characteristics are utilized toestablish a constant gain ratio between the fluorescence and referenceimage signal paths. Of course, when calibrating the light response of afluorescence endoscopy video system, the entire signal path must beconsidered. For simplicity, in this embodiment the gains applied to theimage signals over the remainder of the image signal path (i.e.,excluding the image sensors) are adjusted and are fixed so as not tocontribute to the ratio of the overall image signal gains. As a result,maintaining a constant system image signal gain ratio is reduced toestablishing a constant gain ratio between the two image sensors.

A method for calibrating the gain response of the fluorescence andreference image sensors will now be described. The particulars of thecalibration method depend on the types of sensors utilized. Thecalibration method described herein is for the preferred sensor types:an ICCD for the low light image sensor 104, and a color CCD for thecolor image sensor 102.

The gain of an ICCD sensor (K_(ICCD)) is typically controlled by varyingan analog gain control signal (G). (Such a gain control signal operateson the accelerating voltage that controls the light signal amplificationin the intensifier's multichannel plate.) In such sensors, the gain canbe varied over about four orders of magnitude of light intensity. Thegain/control voltage relationship is approximately exponential and canbe characterized by K_(ICCD))=K₀·e^(f) ^(ICCD) ^((G)) where K₀ is theoverall gain of the ICCD with the gain control setting at zero, andf_(ICCD)(G)=a₁·G+a₂·G²+a₃·G³ is a quasilinear function approximated by apolynomial whose coefficients a_(i) are determined by empiricalmeasurements of the response of the ICCD with varying gain.

The gain of a color CCD can be controlled in two ways: 1) by changingthe electronic shutter time (typically in discrete steps) which allowsvariation in sensitivity over about three orders of magnitude in lightintensity, and 2) by changing an analog electronic gain control whichallows variation in sensitivity over about one order of magnitude inlight intensity. For a CCD, the analog electronic gain typically variesexponentially with a control voltage (R). The gain response of a CCD isthus K_(CCD)=K₆₀·A_(shutter)·^(f) ^(CCD) ^((R)), where K₆₀ is theoverall CCD gain with the electronic shutter at the standard video fieldrate (e.g., 1/60 second for NTSC video) and with the control voltage setto zero, A_(shutter) is the attenuation provided by the electronicshutter, and f_(CCD)(R)=b_(r1)·R+b_(r2)·R²+b_(r3)·R³ is a quasilinearfunction approximated by a polynomial whose coefficients b_(i) aredetermined by empirical measurements of the CCD response with varyinggain. The gain of the CCD can be adjusted to accommodate a wide range inlight intensity by varying A_(shutter), which provides step-wisevariation over a wide range, in combination with R, which allowscontinuous variation over a small range.

To maintain a constant relative light signal response from the imagesensors, the following gain ratio is maintained constant:

$\begin{matrix}{\frac{K_{ICCD}}{K_{CCD}} = {\frac{K_{0} \cdot ^{f_{ICCD}{(G)}}}{K_{60} \cdot A_{shutter} \cdot ^{f_{CCD}{(R)}}} = {{const}.}}} & (1)\end{matrix}$

This constant gain ratio can be implemented by designating one imagesensor as the “master.” For a given gain setting of the “master” imagesensor, the gain setting of the other image sensor (the “slave”) isdetermined by solving Equation 1 to find the appropriate value of R,A_(shutter) (or G). Either image sensor may be utilized as the master.The choice as to which image sensor is utilized as the master and whichthe slave depends on factors such as which image signal predominates inthe digital domain of the image processor, the technique for solving theequation, and on the time it takes each image sensor to respond to achange in gain.

The gain calibration method required for other types of image sensorsutilizes the same principles, including starting with an equationdescribing the gain of each sensor in terms of controllable parameters,calculating the ratio of the gain equations, assuming the gain ratio isconstant, and solving the gain ratio equation for the parameters of onesensor in terms of the parameters of the other sensor and the constant,and can be derived in a similar manner.

In fluorescence/fluorescence mode, the operation of the system issimilar to that of fluorescence/reflectance mode, so only the points ofdifference will be described. Firstly, the light source 52 selects andpositions the appropriate optical filter 76C into the optical pathbetween the arc lamp 70 and the endoscope light guide 54. This filter76C transmits substantially those wavelengths of light that will inducethe tissue 58 under examination to fluoresce.

The autofluorescence emitted by the tissue 58 is collected by theendoscope image guide 56 and is projected through the camera beamsplitter 106 onto the low light image sensor 104 and the color imagesensor 102. Spectral filter 118 limits the light transmitted to the lowlight image sensor 104 to either green or red autofluorescence lightonly and excludes light in the excitation waveband. Spectral filter 119is inserted into the optical path to the color image sensor 102 duringthis imaging mode and transmits only the autofluorescence light in thewaveband not transmitted to the low light image sensor 104. (Note thatspectral filter 119 and, if required, glass block 121, cannot be locatedbetween the beam splitter 106 and the endoscope 60 for this mode ofoperation.) The autofluorescence images are then transduced by the lowlight image sensor 104 and the color image sensor 102 and the resultingimage signals are transmitted to the processor/controller 64. Afterbeing processed, the images from the two sensors are combined into asingle fluorescence/fluorescence image, which is displayed on the videomonitor 66. The image sensor gains are controlled in the same calibratedfashion as for fluorescence/reflectance imaging.

Since the autofluorescence image detected with the color image sensor102 will be very dim, the images obtained with this type of sensor willlikely not be acquired, processed and displayed in real-time unless someform of signal amplification (e.g., pixel binning, CCD with chargecarrier multiplication, etc.) is provided. Currently, it is alsopossible to combine a time-averaged image from the color image sensor102 with a real-time image from the low light image sensor 104 and thendisplay the resulting combined image. Alternatively, images from bothsensors could be time-averaged and combined before being displayed.

A second embodiment of this invention will now be described. All pointsof similarity with the first embodiment will be assumed understood andonly points that differ will be described.

In this second embodiment, all aspects of the system are similar tothose of the first embodiment except the camera 100A. The camera 100Bfor this embodiment of a fluorescence endoscopy video system is as shownin FIG. 5. It differs from the camera in the first embodiment in thatall imaging modes utilize a single, high sensitivity color image sensor102A, preferably a CCD with charge carrier multiplication, a three-CCDimage sensor assembly with charge carrier multiplication, a color CCD, athree-CCD color image sensor assembly, a color CMOS image sensor, or athree-CMOS color image sensor assembly.

In this embodiment, two images are projected onto the sensor 102Asimultaneously. The images are separated and processed by the imageprocessor 64 and displayed according to the imaging mode of the system.In color imaging mode, the color image is separated from the otherimages, processed and displayed on the video monitor 66. For the colorimaging mode, filter 119 is moved out of the light path and glass block121, if required, is moved into position. For fluorescence/reflectanceand fluorescence/fluorescence imaging modes, the fluorescence andreference images are first separated by the image processor 64,processed, and then are again superimposed on the video monitor 66 byapplying each image to a different monitor color input.

A direct consequence of using a single high sensitivity color imagesensor, as described in this embodiment, is that the gain of thefluorescence and reference images automatically track each other as thegain of the sensor is changed. The gain ratio of the two image signalsis determined and maintained by the transmission characteristics offilters 118 and 119 in the camera, and 76B or 76C in the light source.The image processor 64 may also be utilized to implement small changesin the gain ratio by changing the brightness of one image with respectto the other during processing.

As mentioned previously, the autofluorescence images detected with thecolor image sensor 102A will be very dim, and so the images obtainedwith this type of sensor will likely not be acquired, processed, anddisplayed in real-time unless some form of signal amplification (e.g.,pixel binning, color CCD with charge carrier multiplication, etc.) isprovided. Alternatively, the camera may be used to imageautofluorescence in a non-real time mode.

This configuration of the camera also adds an additional restriction tothe design of the optical subsystem. The effect of this restrictionnecessitates that either imaging optical component 112 differs fromimaging optical component 114 in such a way that both images areprojected onto the same image plane, or that beam splitter 106, aftersplitting the light from the endoscope 60, utilizes substantially equaloptical path lengths for both beams and, in conjunction with similarimaging optical components 112 and 114, projects both images onto thesame image plane. Such a beam splitter 106 requires a multicomponent orcustom beam splitter 106 of the type shown in FIGS. 4D-E. The beamsplitters shown in these drawings also anticipate the need for an equaloptical path length, as described for the imaging optics configurationin FIG. 3C.

A third embodiment of this invention will now be described. All pointsof similarity with the first embodiment will be assumed understood andonly points that differ will be described.

In this third embodiment, all aspects of the system are similar to thoseof the first embodiment except the camera 100A. The camera 100C for thisembodiment of a fluorescence endoscopy video system is as shown in FIG.6. It differs from the camera 100A in the first embodiment in that thecolor image sensor 102 is utilized only for the color imaging mode. As aconsequence, filter 119 has been removed from the color image sensoroptical path, which also eliminates the need for a filter movingmechanism. Instead, the light that is not being projected towards thecolor image sensor 102 after being split by the beam splitter 106 isprojected towards a dichroic splitting and filtering assembly 120. Thisassembly 120 further splits and filters the light from the beam splitter106 into two spectral components.

Rather than splitting the incoming light into two beams with the samespectrum but a fractional intensity of the incoming light, a dichroicsplitter 120 divides the incoming light spectrally, so that certainwavelengths are reflected while others are transmitted. Furtherfiltering may then be applied to this spectrally divided light beam.

Several possible configurations for such a dichroic splitting andfiltering assembly 120 are shown in FIG. 7. As shown in the figure, thedichroic splitting and filtering assembly 120 may comprise a cubedichroic 130 or a plate dichroic 133. Spectral filters 118, 119 may bepositioned away from the dichroic mirrors or, in the case of the cube,may be formed as a coating on the cube. In addition, with eitherembodiment, a reflecting mirror 140 may be used to invert the imagereflected off the dichroic mirror. In addition, the dichroic splittermay be configured as a custom prism assembly as shown in FIG. 9.

It should again be noted that if the optical path between the endoscope60 and image sensors contains an uneven number of reflections (e.g.,such as from a single component beam splitter or dichroic), the imageprojected onto the sensor will be left-to-right inverted. Theorientation of such images will need to be corrected by imageprocessing.

After exiting the assembly 120, one of the spectral components isprojected onto the low light image sensor 104 and the second componentis projected onto a separate reference sensor 105. The reference sensor105 preferably comprises a monochrome CCD, monochrome CCD with chargecarrier multiplication, ICCD, CID, CMD, CMOS or EBCCD-type sensor, butit may also be a color CCD, a three-CCD color image sensor assembly, acolor CCD with charge carrier multiplication, a three-color CCD imagesensor assembly with charge carrier multiplication, a color CMOS imagesensor, or a three-CMOS color image sensor assembly. In the case of acolor image sensor, depending on the sensitivity of the sensor,autofluorescence images obtained will likely not be acquired, processedand displayed in real-time unless some form of signal amplification(e.g., pixel binning, CCD with charge carrier multiplication, etc.) isprovided. Alternatively, for fluorescence/fluorescence mode operation,the camera may combine a real-time autofluorescence image (from the lowlight image sensor 104) with a time-averaged image from the referencedsensor 105, or may provide all autofluorescence images in non-real timemode.

Calibration of the light signal path for this embodiment is similar tothat of the first embodiment for the preferred choice of image sensors,in which an ICCD is the low light image sensor 104 and a CCD is thereference image sensor 105. For the case in which the reference imagesensor is also an intensified sensor such as an ICCD or EBCCD, theequation describing the gain ratio for the two sensors is slightlydifferent.

As mentioned above, the gain/control voltage characteristics of an ICCD(or EBCCD) image sensor is approximately exponential and can becharacterized by K_(ICCD)=K₀·e^(f) ^(ICCD) ^((G)), where K₀ is theoverall gain of the ICCD with the gain control setting at zero, G is theintensifier gain signal, and f_(ICCD)(G)=a₁·G+a₂·G²+a₃·G³ is aquasilinear function approximated by a polynomial whose coefficientsa_(i) are determined by empirical measurements of the response of theICCD with varying gain.

With two ICCDs, the gain ratio to be maintained constant is

$\begin{matrix}{\frac{{K_{ICCD}}_{fluor}}{K_{{ICCD}_{ref}}} = {\frac{K_{0_{fluor}} \cdot ^{f_{ICCDfluor}{(G_{fluor})}}}{K_{0_{ref}} \cdot ^{f_{{ICCD}_{ref}}{(G_{ref})}}} = {{const}.}}} & (2)\end{matrix}$

As described in previous embodiments, the gain setting G_(fluor) (orG_(ref)) of one image sensor (the “master”) is determined by anautomatic gain control. The gain setting of the other image sensor (the“slave”) is determined by solving Equation 2 to find the appropriatevalue of G_(ref) (or G_(fluor)). As discussed previously, either imagesensor may be utilized as the master.

A fourth embodiment of this invention will now be described. All pointsof similarity with the third embodiment will be assumed understood andonly points that differ will be described.

In this fourth embodiment, all aspects of the system are similar tothose of the third embodiment except the camera 100C. The camera 100Dfor this embodiment of a fluorescence endoscopy video system is as shownin FIG. 8. It differs from the camera 100C in the third embodiment inthat the low light image sensor 104 is utilized to image both the firstfluorescence image as well as the reference fluorescence or reflectanceimage.

As with the configuration of the beam splitter 106 in the secondembodiment, the configurations of the dichroic splitter and filterassembly 120 and, if necessary, in combination with imaging opticalcomponents 114A and 114B, project both the primary fluorescence and thereference image into the same image plane.

To project the light that passes through the dichroic mirror and thelight that is reflected off the dichroic mirror in the same plane, thedichroic assembly 120 may include a right angle prism 131 and a glassblock 132 that compensate for the differing optical path lengths asshown in FIG. 9A. Alternatively, as shown in FIG. 9B, the dichroicassembly 120 may include a number of prisms having partially and fullyreflective surfaces in the same configured manner as the beam splittershown in FIG. 4E, except that the partially reflecting surface 95 isreplaced with a dichroic mirror surface. In another alternative, theimaging optical component 114A differs from imaging optical component114B in such a way that both images are projected onto the same imageplane.

When using the camera shown in FIG. 8 for fluorescence/reflectanceimaging, the transmission of the filter used for the referencereflectance image (e.g., 114B) and light source filter 76B in FIG. 2 ischosen in such a way that the intensity of the reference reflected imageat sensor 104 is similar to that of the fluorescence image for allpossible excitation light intensities. Also in similar fashion to thatdescribed for the second embodiment, the images transduced by the lowlight image sensor 104 are separated by the image processor 64, areprocessed, and then are again superimposed on the video monitor 66 byapplying each image to a different monitor color input. A fluorescenceendoscopy video system utilizing this embodiment is calibrated in asimilar manner to that described in the second embodiment to maintainconstant gain ratio.

A fifth embodiment of this invention will now be described. All pointsof similarity with the first embodiment will be assumed understood andonly points that differ will be described.

In this fifth embodiment, all aspects of the system are similar to thoseof the first embodiment except the camera 100A. The camera 100E for thisembodiment of a fluorescence endoscopy video system is as shown in FIG.10. It differs from the camera 100A in the first embodiment in that allimaging modes utilize a single, high sensitivity color image sensor102A. It differs from the camera in the second embodiment in that thebeam splitter is removed and the need for spectral filters 118 and 119is eliminated. Each of the pixel elements on the high sensitivity colorsensor 102A is covered by an integrated filter, typically red, green, orblue. These filters block the reflected excitation light and allow thefluorescence and reflectance light to reach the pixel elements.Alternatively, if it is not possible to achieve sufficient blocking ofthe excitation light by means of filters on the color image sensor, aseparate blue blocking filter 118′ can be provided. The blue blockingfilter 118′ is a long pass filter that blocks light at blue and shorterwavelengths and transmits light at green and longer wavelengths. Whensuch a blue blocking filter 118′ is utilized, the intensity of thereflected excitation light is reduced to the point that the integratedfilters on the pixel elements provide sufficient further filtering todefine the wavelengths of fluorescence and reflectance light that reachthe high sensitivity color sensor 102A.

In this embodiment, the primary fluorescence and reference images aresuperimposed over the same area of the image sensor 102A but, because ofthe individual filters placed over each pixel, these images are detectedby different sensor pixels. Separate primary fluorescence and referenceimage signals can then be created by the image processor 64 from thesingle CCD image signal.

In the color imaging mode, if it is utilized for fluorescence imaging,the blue blocking filter 118′ is removed from the light path and, ifrequired, glass block 121 is moved into position. The color image isprocessed by image processor 64 and displayed on the video monitor 66.For fluorescence/reflectance and fluorescence/fluorescence imaging modesthe fluorescence and reference images are processed by image processor64 and superimposed on the video monitor 66 by applying each image to adifferent color input of the monitor. The way in which this embodimentis calibrated to maintain constant relative gain is similar to thatdescribed for the second embodiment.

The reference light transmission specifications of both the light sourcefilter 76B or 76C and the selective color filters integrated with theimage sensor 102A are chosen such that the intensity of the reflectedlight at the color image sensor active elements results in a transducedimage signal with good signal-to-noise characteristics and withoutsignificant saturation. At the same time these filters must haveappropriate light transmission specifications for excitation and imagingof the primary fluorescence. The filter transmission characteristicsmust further be chosen to provide the desired ratio of relative primaryfluorescence to reference light intensity at the image sensor.

As mentioned previously, the autofluorescence images detected with thecolor image sensor will be very dim, and so the images obtained withthis type of sensor will likely not be acquired, processed and displayedin real-time unless some form of signal amplification (e.g., pixelbinning, CCD with charge carrier multiplication, etc.) is provided.Alternatively, the camera may be used to image autofluorescence innon-real time mode.

As will be appreciated, each of the embodiments of the camera describedabove are lighter in weight than prior art because no more than one lowlight image sensor 104 is required. Since such sensors are often heavy,bulky and expensive, the size and cost of the camera is significantlyreduced. Furthermore, because a fixed beam splitter 106 is used insteadof a movable mirror, the cameras are more robust and can be made lessexpensively.

As indicated above, the filters in the light source and camera should beoptimized for the imaging mode of the camera, the type of tissue to beexamined, and/or the type of pre-cancerous tissue to be detected.Although all of the filters described below can be obtained made toorder using standard, commercially available components, the appropriatewavelength range of transmission and degree of blocking outside of thedesired transmission range for the described fluorescence endoscopyimages modes are important to the proper operation of the system. Theimportance of other issues in the specification of such filters such asthe fluorescence properties of the filter materials and the proper useof anti-reflection coatings are taken to be understood.

FIGS. 11-14 illustrate the preferred filter characteristics for use in afluorescence endoscopy video system operating influorescence/reflectance imaging mode wherein both tissueautofluorescence is being excited and imaged and a reference reflectancelight is being reflected and imaged. There are several possibleconfigurations of fluorescence endoscopy video systems, operating in thefluorescence/reflectance imaging mode including green fluorescence witheither red or blue reflectance, and red fluorescence with either green,blue, or near-infrared reflectance. The particular configurationutilized depends on the target clinical organ and application. Thefilter characteristics will now be described for each of these fourconfigurations.

FIGS. 11A-11D illustrate the preferred composition of the lighttransmitted by filters for a green fluorescence and red reflectanceimaging mode. FIG. 11A illustrates the composition of the lighttransmitted by the light source filter, such as filter 76B, which isused to produce blue excitation light and red reference light. Thisfilter transmits light in the blue wavelength range from 370-460 nm, orany subset of wavelengths in this range. It also transmits light in thered wavelength range of 590-750 nm, or any subset of wavelengths in thisrange. The light transmitted in the red wavelength range (or subset ofthat range) is adjusted, as part of the system design, to be anappropriate fraction of the light transmitted in the blue wavelengthrange. This fraction is selected to meet the need to match the intensityof the reflected reference light projected on the color image sensor tothe requirements of the sensor, at the same time as maintainingsufficient fluorescence excitation. Of the light transmitted by thisfilter, less than 0.001% is in the green wavelength range of 480-570 nm(or whatever desired subset of this range is specified as thetransmission range of the green fluorescence filter described below).

FIG. 11B shows the composition of the light transmitted by a camerafilter, such as spectral filter 118, for imaging the green fluorescenceimage. In this configuration, the filter blocks the blue excitationlight and red reflectance light while transmitting green fluorescencelight in the wavelength range of 480-570 nm, or any subset ofwavelengths in this range. When used in a fluorescence endoscopy videosystem with the light source filter 76B described above and the dichroicmirror described below, the filter characteristics are such that anylight outside of the wavelength range of 480-570 nm (or any desiredsubset of wavelengths in this range) contributes no more than 0.1% tothe light transmitted by the filter.

FIG. 11C shows the composition of the light transmitted by a camerafilter, such as spectral filter 119, for imaging the red reflectanceimage. In this configuration, the filter blocks the blue excitationlight and green fluorescence light while transmitting red reflectancelight in the wavelength range of 590-750 nm, or any desired subset ofwavelengths in this range. When used in a fluorescence endoscopy videosystem with the light source filter 76B described above and the dichroicmirror described below, the filter characteristics are such that anylight outside of the wavelength range of 590-750 nm (or any desiredsubset of wavelengths in this range) contributes no more than 0.1% tothe light transmitted by the filter. If the reference image sensor is acolor image sensor, such as a color CCD, then further filtering may beobtained from the color filters integrated with the sensor. The in-bandtransmission characteristics (in the wavelength range of 590-750 nm, orany desired subset of wavelengths in this range) are determined by theneed to match the intensity of the reflected reference light projectedonto the color image sensor to the requirements of the sensor, incombination with the characteristics of the light source filterdescribed above.

FIG. 11D shows the composition of the light transmitted by a dichroicmirror of the kind that may be employed in the dichroic splitter andfilter assembly 120. The dichroic mirror preferably has a half-maximumtransmission in the range of 570-590 nm. It may reflect the shorterwavelengths and transmit the longer wavelengths (long pass) or transmitshorter wavelengths and reflect longer wavelengths (short pass). Asdescribed above, the dichroic splitter and filter assembly mayincorporate the filters shown in FIGS. 11B and 11C.

FIGS. 12A-12D illustrate the preferred composition of the lighttransmitted by filters for a green fluorescence and blue reflectanceimaging mode. FIG. 12A illustrates the composition of the lighttransmitted by a light source filter which is used to produce excitationlight, such as filter 76B described above. In the case of afluorescence/reflectance imaging mode utilizing blue reflectance, thewavelengths of the imaged reflectance light are contained within therange of blue excitation wavelengths. The filter transmits light in thewavelength range from 370-460 nm, or any subset of wavelengths in thisrange, but it is not required to transmit any light in the redwavelength range. Of the light transmitted by this filter, less than0.001% is in the green wavelength range of 480-570 nm (or whateverdesired subset of this range is specified as the transmission range ofthe green fluorescence filter described below).

FIG. 12B shows the composition of the light transmitted by a camerafilter for imaging the green fluorescence image, such as spectral filter118. The composition of the light transmitted by this filter has thesame characteristics as the light described in FIG. 11B.

FIG. 12C shows the composition of the light transmitted by a camerafilter, such as filter 119, for imaging the blue reflectance image. Inthis configuration, the filter blocks the green fluorescence light whiletransmitting blue reflectance light in the wavelength range of 370-460nm, or any desired subset of wavelengths in this range. Depending on thesensitivity of the image sensor used to transduce the blue reflectanceimage, the transmission of this filter may need to be restricted so asto prevent the large amount of reflected blue light from overwhelmingthe sensor. When used in a fluorescence endoscopy video system with thelight source filter 76B described above and the dichroic mirrordescribed below, the filter characteristics are such that any lightoutside of the wavelength range of 370-460 nm, or any desired subset ofwavelengths in this range, contributes no more than 0.1% to the lighttransmitted by the filter. If the reference image sensor is a colorimage sensor, such as a color CCD, then further filtering of thereflected blue light may be obtained from the color filters integratedwith the sensor.

FIG. 12D shows the composition of the light transmitted by a dichroicmirror of the kind that may be employed in the dichroic splitter andfilter assembly 120. The dichroic mirror preferably has a half-maximumtransmission in the range of 460-480 nm. It may reflect the shorterwavelengths and transmit the longer wavelengths (long pass) or transmitshorter wavelengths and reflect longer wavelengths (short pass). Asdescribed above the dichroic splitter and filter assembly mayincorporate the filters shown in FIGS. 12B and 12C.

FIGS. 13A-13D illustrate the preferred composition of the lighttransmitted by filters for a red fluorescence and blue reflectanceimaging mode. FIG. 13A illustrates the composition of the lighttransmitted by a light source filter, such as filter 76B, which is usedto produce blue excitation light. This filter transmits light in thewavelength range from 370-460 nm, or any subset of wavelengths in thisrange. Of the light transmitted by this filter, less than 0.001% is inthe red fluorescence imaging wavelength range of 590-750 nm (or whateverdesired subset of this range is specified as the transmission range ofthe red fluorescence filter described below).

FIG. 13B shows the composition of the light transmitted by a camerafilter, such as spectral filter 118, for imaging the red fluorescenceimage. In this configuration, the filter blocks the blue excitationlight, while transmitting red fluorescence light in the wavelength rangeof 590-750 nm, or any subset of wavelengths in this range. When used ina fluorescence endoscopy video system with the light source filter 76Bdescribed above and the dichroic mirror described below, the filtercharacteristics are such that any light outside of the wavelength rangeof 590-750 nm, or any desired subset of wavelengths in this range,contributes no more than 0.1% to the light transmitted by the filter.

FIG. 13C shows the composition of the light transmitted by a camerafilter, such as filter 119, for imaging the blue reflectance image. Thecomposition of the light transmitted by this filter has the samecharacteristics as the light described in FIG. 12C.

FIG. 13D shows the composition of the light transmitted by a dichroicmirror of the kind that may be employed in the dichroic splitter andfilter assembly 120 to split the red fluorescence and blue reflectance.The dichroic mirror preferably has a half-maximum transmission in therange of 460-590 nm. It may reflect the shorter wavelengths and transmitthe longer wavelengths (long pass) or transmit shorter wavelengths andreflect longer wavelengths (short pass). As described above the dichroicsplitter and filter assembly may incorporate the filters described inFIGS. 13B and 13C.

FIGS. 14A-14D illustrate the preferred composition of the lighttransmitted by filters for a red fluorescence and green reflectanceimaging mode. FIG. 14A illustrates the composition of the lighttransmitted by a light source filter which is used to produce excitationlight, such as filter 76B described above. This filter transmits lightin the blue wavelength range from 370-460 nm, or any subset ofwavelengths in this range. It also transmits light in the greenwavelength range of 480-570 nm, or any subset of wavelengths in thisrange. The light transmitted in the green wavelength range (or subset ofthat range) is adjusted, as part of the system design, to be anappropriate fraction of the light transmitted in the blue wavelengthrange. This fraction is selected to meet the need to match the intensityof the reflected reference light projected on the color image sensor tothe requirements of the sensor, at the same time as maintainingsufficient fluorescence excitation. Of the light transmitted by thisfilter, less than 0.001% is in the red fluorescence imaging wavelengthrange of 590-750 nm (or whatever desired subset of this range isspecified as the transmission range of the red fluorescence filterdescribed below).

FIG. 14B shows the composition of the light transmitted by a camerafilter, such as spectral filter 118, for imaging the red fluorescenceimage. In this configuration, the filter blocks the blue excitationlight and green reflectance light while transmitting red fluorescencelight in the wavelength range of 590-750 nm, or any subset ofwavelengths in this range. When used in a fluorescence endoscopy videosystem with the light source filter 76B described above and the dichroicmirror described below, the filter characteristics are such that anylight outside of the wavelength range of 590-750 nm, or any desiredsubset of wavelengths in this range, contributes no more than 0.1% tothe light transmitted by the filter.

FIG. 14C shows the composition of the light transmitted by a camerafilter, such as filter 119, for imaging the green reflectance image. Inthis configuration, the filter blocks the blue excitation light and redfluorescence light while transmitting green reflectance light in thewavelength range of 480-570 nm, or any desired subset of wavelengths inthis range. The in-band transmission characteristics (in the wavelengthrange of 480-570 nm, or any desired subset of wavelengths in this range)are determined by the need to match the intensity of the reflectedreference light projected onto the color image sensor to therequirements of the sensor, in combination with the characteristics ofthe light source filter described above. When used in a fluorescenceendoscopy video system with the light source filter 76B described aboveand the dichroic mirror described below, the filter characteristics aresuch that any light outside of the wavelength range of 480-570 nm, orany desired subset of wavelengths in this range, contributes no morethan 0.1% to the light transmitted by the filter.

FIG. 14D shows the composition of the light transmitted by a dichroicmirror of the kind that may be employed in the dichroic splitter andfilter assembly 120 to split the red fluorescence and green reflectance.The composition of the light transmitted by this filter has the samecharacteristics as the light described in FIG. 11D.

FIGS. 15A-15D illustrate the preferred composition of the lighttransmitted by filters for a red fluorescence and near-infraredreflectance imaging mode. FIG. 15A illustrates the composition of thelight transmitted by a light source filter, which is used to produceexcitation light such as filter 76B described above. This filtertransmits light in the blue wavelength range from 370-460 nm, or anysubset of wavelengths in this range. It also transmits light in thenear-infrared wavelength range of 700-850 nm, or any subset ofwavelengths in this range. The light transmitted in the near-infraredwavelength range (or subset of that range) is adjusted, as part of thesystem design, to be an appropriate fraction of the light transmitted inthe blue wavelength range to meet the need to match the intensity of thereflected reference light projected on the color image sensor to therequirements of the sensor, at the same time as maintaining sufficientfluorescence excitation. Of the light transmitted by this filter, lessthan 0.001% is in the red fluorescence imaging wavelength range of590-700 nm (or whatever desired subset of this range is specified as thetransmission range of the red fluorescence filter described below).

FIG. 15B shows the composition of the light transmitted by a camerafilter, such as spectral filter 118, for imaging the red fluorescenceimage. In this configuration, the filter blocks the blue excitationlight and near-infrared reflectance light while transmitting redfluorescence light in the wavelength range of 590-700 nm, or any subsetof wavelengths in this range. When used in a fluorescence endoscopyvideo system with the light source filter 76B described above and thedichroic mirror described below, the filter characteristics are suchthat any light outside of the wavelength range of 590-700 nm, or anydesired subset of wavelengths in this range, contributes no more than0.1% to the light transmitted by the filter.

FIG. 15C shows the composition of the light transmitted by a camerafilter, such as filter 119, for imaging the near-infrared reflectanceimage. In this configuration, the filter blocks the blue excitationlight and red fluorescence light while transmitting near-infraredreflectance light in the wavelength range of 700-850 nm, or any desiredsubset of wavelengths in this range. The in-band transmissioncharacteristics (in the wavelength range of 700-850 nm, or any desiredsubset of wavelengths in this range) are determined by the need to matchthe intensity of the reflected reference light projected onto the colorimage sensor to the requirements of the sensor, in combination with thecharacteristics of the light source filter described above. When used ina fluorescence endoscopy video system with the light source filter 76Bdescribed above and the dichroic mirror described below, the filtercharacteristics are such that any light outside of the wavelength rangeof 700-850 nm, or any desired subset of wavelengths in this range,contributes no more than 0.1% to the light transmitted by the filter.

FIG. 15D shows the composition of the light transmitted by a dichroicmirror of the kind that may be employed in the dichroic splitter andfilter assembly 120 to split the red fluorescence and near-infraredreflectance. The dichroic mirror preferably has a half-maximumtransmission in the range of 690-710 nm. It may reflect the shorterwavelengths and transmit the longer wavelengths (long pass) or transmitshorter wavelengths and reflect longer wavelengths (short pass). Asdescribed above the dichroic splitter and filter assembly mayincorporate the filters described in FIGS. 15B and 15C.

FIGS. 16A-16D illustrate the preferred composition of the lighttransmitted by filters for a green fluorescence and near-infraredreflectance imaging mode. FIG. 16A illustrates the composition of thelight transmitted by a light source filter which is used to produceexcitation light, such as filter 76B described above. This filtertransmits light in the blue wavelength range from 370-460 nm, or anysubset of wavelengths in this range. It also transmits light in thenear-infrared wavelength range of 700-850 nm, or any subset ofwavelengths in this range. The light transmitted in the near-infraredwavelength range (or subset of that range) is adjusted, as part of thesystem design, to be an appropriate fraction of the light transmitted inthe blue wavelength range to meet the need to match the intensity of thereflected reference light projected on the color image sensor to therequirements of the sensor, at the same time as maintaining sufficientfluorescence excitation. Of the light transmitted by this filter, lessthan 0.001% is in the green fluorescence imaging wavelength range of480-570 nm (or whatever desired subset of this range is specified as thetransmission range of the red fluorescence filter described below).

FIG. 16B shows the composition of the light transmitted by a camerafilter, such as spectral filter 118, for imaging the green fluorescenceimage. In this configuration, the filter blocks the blue excitationlight and near-infrared reflectance light while transmitting greenfluorescence light in the wavelength range of 480-570 nm, or any subsetof wavelengths in this range. When used in a fluorescence endoscopyvideo system with the light source filter 76B described above and thedichroic mirror described below, the filter characteristics are suchthat any light outside of the wavelength range of 480-570 nm, or anydesired subset of wavelengths in this range, contributes no more than0.1% to the light transmitted by the filter.

FIG. 16C shows the composition of the light transmitted by a camerafilter, such as filter 119, for imaging the near-infrared reflectanceimage. In this configuration, the filter blocks the blue excitationlight and green fluorescence light while transmitting near-infraredreflectance light in the wavelength range of 700-850 nm, or any desiredsubset of wavelengths in this range. The in-band transmissioncharacteristics (in the wavelength range of 700-850 nm or any desiredsubset of wavelengths in this range) are determined by the need to matchthe intensity of the reflected reference light projected onto the colorimage sensor to the requirements of the sensor, in combination with thecharacteristics of the light source filter described above. When used ina fluorescence endoscopy video system with the light source filter 76Bdescribed above and the dichroic mirror described below, the filtercharacteristics are such that any light outside of the wavelength rangeof 700-850 nm, or any desired subset of wavelengths in this range,contributes no more than 0.1% to the light transmitted by the filter.

FIG. 16D shows the composition of the light transmitted by a dichroicmirror of the kind that may be employed in the dichroic splitter andfilter assembly 120 to split the green fluorescence and near-infraredreflectance. The dichroic mirror preferably has a half-maximumtransmission in the range of 590-660 nm. It may reflect the shorterwavelengths and transmit the longer wavelengths (long pass) or transmitshorter wavelengths and reflect longer wavelengths (short pass). Asdescribed above, the dichroic splitter and filter assembly mayincorporate the filters described in FIGS. 16B and 16C.

FIGS. 17A-17D illustrate the preferred composition of the lighttransmitted by filters for use in a fluorescence endoscopy video systemoperating in fluorescence/fluorescence imaging mode wherein the tissueautofluorescence being excited and imaged is divided into two spectralbands.

FIG. 17A illustrates the composition of the light transmitted by afilter, such as filter 76C, which is used to produce excitation light inthe system light source. This filter transmits light in the wavelengthrange from 370-460 nm, or any subset of wavelengths in this range. Ofthe light transmitted by this filter, less than 0.001% is in thefluorescence imaging band from 480-750 nm (or whatever desired subsetsof this range are within the specified transmission range of the primaryand reference fluorescence image filters described below).

FIG. 17B shows the composition of the light transmitted by a camerafilter, such as filter 118, for imaging the primary fluorescence image.In this configuration, the filter blocks excitation light and redfluorescence light while transmitting green fluorescence light in thewavelength range of 480-570 nm, or any subset of wavelengths in thisrange. When used in a fluorescence endoscopy video system with the lightsource filter 76C described above and the dichroic mirror describedbelow, the filter characteristics are such that any light outside of thewavelength range of 480-570 nm, or any desired subset of wavelengths inthis range, contributes no more than 0.1% to the light transmitted bythe filter.

FIG. 17C shows the composition of the light transmitted by a camerafilter for imaging the reference fluorescence image, such as filter 119.In this configuration, the filter blocks excitation light and greenfluorescence light while transmitting red fluorescence light in thewavelength range of 590-750 nm, or any subset of wavelengths in thisrange. When used in a fluorescence endoscopy video system with the lightsource filter 76C described above and the dichroic mirror describedbelow, the filter characteristics are such that any light outside of thewavelength range of 590-750 nm, or any desired subset of wavelengths inthis range, contributes no more than 0.1% to the light transmitted bythe filter.

FIG. 17D shows the composition of the light transmitted by a dichroicmirror of the kind that may be employed in the dichroic splitter andfilter assembly 120. The dichroic mirror preferably has a half-maximumtransmission in the range of 570-590 nm. It may reflect the shorterwavelengths and transmit the longer wavelengths (long pass) or transmitshorter wavelengths and reflect longer wavelengths (short pass).

FIG. 18 shows the composition of light transmitted by a filter 118′employed for blocking blue light in a camera such as that described inthe fifth embodiment and shown in FIG. 10. The filter transmits light inthe range 480-750 nm, or any subset of wavelengths of light in thisrange. Of the light transmitted by this filter, less than 0.001% is inthe fluorescence excitation band from 370-460 nm (or whatever desiredsubset of this range is within the specified transmission range of thelight source filters described above).

The fluorescence endoscopy video systems described in the aboveembodiments have been optimized for imaging endogenous tissuefluorescence. They are not limited to this application, however, and mayalso be used for photodynamic diagnosis (PDD) applications. As mentionedabove, PDD applications utilize photoactive drugs that preferentiallyaccumulate in tissues suspicious for early cancer. Since effectiveversions of such drugs are currently in development stages, thisinvention does not specify the filter characteristics that are optimizedfor such drugs. With the appropriate light source and camera filtercombinations, however, a fluorescence endoscopy video system operatingin either fluorescence/fluorescence or fluorescence/reflectance imagingmode as described herein may be used to image the fluorescence from suchdrugs.

Next, an aspect of a fluorescence endoscopy video system containingfeatures to maintain a consistent imaging performance will be described.As mentioned earlier, the light signal response of a fluorescenceendoscopy video system requires calibration. A feature to confirm andmaintain this calibration is essential for clinically effectiveperformance.

FIG. 19 shows a block diagram of the relevant system components involvedin the process of self-calibration. Light from the light source 52 issupplied to an illumination guide 54 of an endoscope 60 and is directedto a fluorescence/reflectance target 59 with known fluorescence andreflectance properties. Depending on the imaging mode, fluorescence andreflectance light from the target 59 is collected and transmittedthrough an image guide 56 of the endoscope to the camera 100. The camera100, operating in the fluorescence/reflectance orfluorescence/fluorescence mode, spectrally splits and transduces imagesinto separate electrical signals, which are then digitized in the imageprocessor/controller 64. The image processor/controller 64 quantifiesthe magnitude of these digitized image signals in terms of image graylevels. By using spatial and temporal averaging, the error in thequantified value of the signal response can be reduced to less than 1%.The image processor/controller 64 then compares the known properties ofthe target to the quantified signal response and adjusts the gain ratiodescribed previously to the desired constant value. This adjustmentcompensates for variations in the signal path between the target 59 andimage processor/controller 64, due to factors such as variations intransmission properties of different endoscopes being used with thesystem and changes in the signal response of the system with age. Suchself-calibration ensures that the gain ratio is set to a value such thattissue suspicious for early cancer in a fluorescence image will appearas a distinctly different color than normal tissue. Thisself-calibration could be carried out before every endoscopy.

Although this method is similar to existing methods used to adjust thecolor response of standard camera systems, such a technique has not beenpreviously applied to multispectral fluorescence orfluorescence/reflectance endoscopy. The method uses a reference target59 that provides suitable known fluorescence and reflectance response tothe light from the light source.

Any suitable object with appropriate fluorescence and reflectanceproperties can be used as a reference target. For example, such areference target 59 can be made by mixing a fluorescent dye(s) and lightscattering materials into a liquid. The liquid used may be a solute(such as methanol) enclosed in a container with an optical window, oralternatively may be a liquid which hardens to form a solid (such as anepoxy). The dye(s) used must be appropriately soluble in the liquidutilized. The fluorescence spectrum and brightness of the target 59 iscontrolled by the choice and concentration of the fluorescence dye (ordyes) contained in the target. The fluorescent dye(s) must be chosensuch that the light emitted by the light source 52 excites fluorescencelight in the green and/or red wave bands defined by the camera filtersdescribed above that correspond to a particular imaging mode. Thefluorescent dye(s) must also be stable with time and not undergosignificant photobleaching. One such fluorescent dye is Coumarin #540A.The concentration of the fluorescence dye in the target is chosen suchthat the emitted fluorescence light produces mid-range signal amplitudesat or near a particular clinically used gain setting.

The reflectance property of the target is controlled by the type andconcentration of scattering material added to the target. The type ofscattering material is chosen for good reflectivity of the referencelight in the wavebands defined by the camera filters described abovethat correspond to a particular fluorescence/reflectance imaging mode.The concentration of the scattering material in the target is chosensuch that the reflected reference light produces mid-range signalamplitudes at or near a particular clinically used gain setting.

Once a reference target having the appropriate fluorescence andreflectance properties has been made, these properties are verified andvalidated using fluorescence spectroscopy and reflectance spectroscopy.

Next, another aspect of a fluorescence endoscopy video system will bedescribed in which the perceived color contrast between normal tissueand tissue suspicious for early cancer is enhanced by means of acontrast enhancement algorithm that is applied to the digitized imagesignals in the image processor/controller 64.

In fluorescence endoscopy video images, the contrast between normaltissue and tissue suspicious for early cancer is typically the result ofa reduction in the fluorescence signal associated with the disease,which is not matched by a corresponding reduction in the referencesignal. Such image areas are therefore characterized by a combination ofreduced image brightness and altered color. In such image areas ofrelative darkness, the color difference between suspected lesions andthe surrounding normal tissue can be difficult to discern. To aidphysicians in detecting these subtle color changes, the presentinvention also includes a method of enhancing the contrast betweennormal and tissue suspicious for early cancer. This method consists of asoftware algorithm that is applied to the digitizedfluorescence/reflectance (or fluorescence/fluorescence) image signals bythe image processor/controller 64, and may be utilized in allembodiments of a fluorescence endoscopy video system describedpreviously.

The contrast enhancement method alters the color and intensity of apixel in the displayed fluorescence video image as a function of thepixel characteristics and, possibly, as a function of the neighboringpixel characteristics. The algorithm consists of a number of elements.Firstly, it characterizes the image on a pixel-by pixel-basis bydetermining properties such as the ratio of the intensity of thereference image to the intensity of the fluorescence image. Thealgorithm may also characterize the image by other properties, such asthe spatial texture associated with the color in an area containing thepixel of interest. In the second step, the algorithm applies a test tothe pixel property values. This test will determine whether the pixelproperty values fall within a certain specified range. Finally, afunction, whose value depends on the results of the test, is applied tochange the pixel display properties. The function changes the propertiesof those pixels whose characterized property values fall within acertain range. These pixels will have their properties changed in such away that, in the displayed video image, they are more easilydistinguished from those pixels that do not have characterized propertyvalues that fall within the specified range. By choosing a test thatselects pixel property values corresponding to early cancer, thecontrast between normal tissue and tissue suggestive for early cancercan be enhanced.

The general algorithm will now be described in more detail. The firststep is to quantify pixel properties. Given that the fluorescence fromtissue areas with early cancer typically exhibits both reducedbrightness and altered color, intensity and color are the pixelproperties that can be used to identify such an area. In a dual imagesensing system, such as those described in the aforementionedembodiments, the algorithm may measure the intensity of the fluorescenceimage, the intensity of the reference image (reflectance orfluorescence), or some combination of these. Since the reference andfluorescence images are acquired in different parts (wavebands) of thefluorescence spectrum, the color of a pixel can be characterized by theratio of the intensity of the reference image to the intensity of thefluorescence image.

Other pixel properties may also be useful in characterizing tissuessuspicious for early cancer. The spatial texture of the color may besuch a property. One means of characterizing the color texture is tocalculate the mean and standard deviation of the ratio of the intensityof the reference image to the intensity of the fluorescence image forpixels in an area of defined size containing the pixel of interest. Thestandard deviation of this ratio provides a measure of the colortexture, which can be associated with the pixel of interest. Another wayto characterize the color texture is to calculate the two-dimensionalFourier transform of the color ratio in an area of defined sizecontaining the pixel of interest. Other pixel or pixel neighborhoodproperties that uniquely characterize tissue suspicious for early cancercan be quantified using similar techniques.

The next step in the algorithm is to apply a test to the values of thepixel properties. Such a test can be single dimensional ormultidimensional. For example, such a test may be based solely on thevalue of one pixel property (e.g., whether or not the ratio of thereference image intensity to the fluorescence image intensity fallswithin a given range) or it may be based on a combination of the valuesof several pixel properties (e.g., whether or not the ratio falls with agiven range and the reference intensity falls within a defined range,and the color texture falls within a given range).

Following the test, a function, which depends on the result of the test,is applied to the properties of the pixel. Such a function changes oneor more pixel properties, based on the outcome of the test. The functioncan operate on both the fluorescence and reference image components ofthe displayed video image or on only one of them. The function can belinear or nonlinear.

Three embodiments of contrast enhancement algorithms for a fluorescenceendoscopy system, of the type described above, will now be illustrated.

The first embodiment of a contrast enhancement algorithm for afluorescence endoscopy system is best described by means of FIG. 20.This figure illustrates the test and corresponding function applied tothe properties of each pixel. The vertical axis in the figure representsthe function 302, a relative gain, to be applied to the digitized imagesignals. A separate gain is applied to the primary fluorescence imagesignal and the reference (reflectance or fluorescence) signal. Thehorizontal axis represents the value of a pixel property 304. In thisembodiment the pixel property 304 is the ratio of the reference(reflectance or fluorescence) image signal (intensity) to the primaryfluorescence image signal.

In the example shown in FIG. 20, the gain applied to the primaryfluorescence image signal is unity. The gain applied to the reference(reflectance or fluorescence) image signal is increased when the ratiofalls within the range defined by break points 306 and 308. As shown inthe figure, the gain function applied to the reference (reflectance orfluorescence) image signal has a constant value up to a break point 302.This gain then increases linearly to a break point 310, continueslinearly to another break point 312, and decreases linearly to breakpoint 308, beyond which it remains constant. The position of the breakpoints on the horizontal axis, and the gain function value at all breakpoints, can be adjusted by the operator of the fluorescence endoscopyvideo system.

It has been determined that, if a fluorescence endoscopy video system isappropriately calibrated as described above, the fluorescence andreflectance image signals from tissues suspicious for early cancer willconsistently and uniquely produce ratio values within a specific range.The operator may select gain break points 306 and 308 to be positionedat the extremes of this range and thereby apply a gain to the referencereflectance (or fluorescence) signal over the entire range of ratiovalues that correspond to tissue suspicious for early cancer.

As described above, the processed primary fluorescence image signal andthe processed reference (reflectance or fluorescence) signal are inputto color video monitor 66 as different color components of a singlesuperimposed image. By selective application of the gain function to thereference (reflectance or fluorescence) signal as described, itscontribution to the color of the superimposed image is increased and thecolor contrast between image pixels of normal tissue and image pixels oftissue suspicious for early cancer is enhanced.

Note that if the piecewise linear function illustrated in FIG. 20 isreplaced by any similar function, not necessarily linear, comparablecontrast enhancement can be obtained.

A second embodiment of the contrast enhancement algorithm will now bedescribed. All points of similarity with the first embodiment will beassumed understood and only points that differ will be described.

In the second embodiment of a contrast enhancement algorithm for afluorescence endoscopy system, in addition to the test and functionoperating on pixel properties described in the first embodiment, asecond additional test and function is applied. The additional testelement and function is illustrated by means of FIG. 21. The verticalaxis in the figure represents the function, a relative gain 322, to beapplied to the digitized image signals. A separate gain function isapplied to the primary fluorescence image signal and the reference(reflectance or fluorescence) signal. The horizontal axis represents thevalue of a pixel property 324, which is either the intensity of theprimary fluorescence image signal, or the intensity of the reference(reflectance or fluorescence) signal, or a two-dimensional combinationof these.

The gain function applied to the fluorescence image signal is unity. Thegain applied to the reference image signal decreases linearly abovebreakpoint 326 to breakpoint 330. It then decreases linearly beyondbreak point 330 to break point 328. Beyond break point 328 the gainfunction is constant. In this embodiment, the tests and functionsillustrated by both FIGS. 20 and 21 are applied sequentially with theresult that two or more sets of gain factors are applied. The net resultis a modification of the intensity value of the pixel of interest by twoor more multiplicative factors applied following two or more separatetests. This embodiment is an example of a multiparameter test discussedpreviously. As in the first embodiment, the operator may select the gainfactor break points shown in FIG. 20. The operator may also select thegain factor break points 326, 328, and 330, along with their associatedgain values. Also as described in the first embodiment, if the piecewiselinear functions illustrated in FIGS. 20 and 21 are replaced by anysimilar functions, not necessarily linear, comparable contrastenhancement can be obtained.

A third embodiment of the contrast enhancement algorithm will now bedescribed. All points of similarity with the first embodiment will beassumed understood and only points that differ will be described.

The third embodiment of a contrast enhancement algorithm for afluorescence endoscopy system is similar to the first embodiment, exceptthat the linear gain function utilized in the first embodiment isreplaced by a nonlinear function. FIG. 22 illustrates the test appliedto the properties of each pixel. This figure is similar to FIG. 20,except that instead of representing gain, the vertical axis representsan intermediate parameter, Q 340. The horizontal axis represents thevalue of a pixel property 304. In this embodiment the pixel property 304is the ratio of the reference (reflectance or fluorescence) image signalvalue to the primary fluorescence image signal value for a given imagepixel. The parameter Q is used to calculate the gain to be applied ateach pixel via Equation 3

$\begin{matrix}{{F\left( r_{in} \right)} = \left( \frac{r_{in}}{r_{\max}} \right)^{\frac{1}{Q} - 1}} & (3)\end{matrix}$

where F(r_(in)) is the gain, r_(in) is the image signal value, andr_(max) is the maximum possible image signal value.

In this embodiment, the value of Q for the primary fluorescence imagesignal is unity for all (reference image signal value to fluorescenceimage signal) ratio values. As a result, the gain calculated from theequation above and applied to the primary fluorescence image signal isalso unity.

The value of Q for the reference image signal increases when the(reference image signal value to fluorescence image signal) ratio fallswithin the range defined by break points 306 and 308. As shown in thefigure, the value of Q has a constant value up to a break point 302,before increasing linearly to a break point 310, continuing linearly toanother break point 312, and decreasing linearly to break point 308,beyond which it remains constant. The position of the break points onthe horizontal axis, and the gain factors at all break points, can beadjusted by the operator of the fluorescence endoscopy video system.

Using the value of Q, the gain function is calculated for each pixel ofthe reference image signal. If the value of Q is greater than one, thevalue of the reference image signal to which the gain is being appliedwill increase nonlinearly with increasing values of Q. The gain appliedto the reference image signal is larger for lower reference image signalvalues. The net result of this test and function is that the resultingcontrast enhancement depends on both the ratio of the reference imagesignal value to the primary fluorescence image signal value and thereference image signal value.

If the piecewise linear function illustrated in FIG. 22 is replaced byany similar function, not necessarily linear, comparable contrastenhancement can be obtained.

While the preferred embodiment of the invention has been illustrated anddescribed, it will be appreciated that various changes can be madetherein without departing from the scope of the invention. It istherefore intended that the scope of the invention be determined fromthe following claims and equivalents thereto.

What is claimed is:
 1. A fluorescence endoscopy video system including:a multimode light source for producing white light, fluorescenceexcitation light, or fluorescence excitation light with a referencereflectance light; an endoscope for directing light from the lightsource to a patient to illuminate a tissue sample and to collect thereflected light or fluorescence light produced by the tissue; a camerapositioned to receive the light collected by the endoscope including: ahigh sensitivity color image sensor having a plurality of pixelelements, each of the pixel elements having an integrated filter, theintegrated filters being configured to block reflected excitation lightfrom reaching the pixel elements and allow fluorescence light andreflectance light to reach the pixel elements, and at least one opticalimaging component that projects images onto the high sensitivity colorimage sensor; a processor that receives image signals from the highsensitivity color image sensor and (i) combines image signals from afirst group of the pixel elements having integrated filters with a firstfilter characteristic to form a first image formed by the fluorescencelight, and (ii) combines image signals from a second group of the pixelelements having integrated filters with a second filter characteristicto form a second image formed by the reflectance light; and a videomonitor for simultaneously superimposing the first image and the secondimage.
 2. The system of claim 1, wherein the camera is attached to theportion of the endoscope that remains outside of the body.
 3. The systemof claim 1, wherein the camera is built into the insertion portion ofthe endoscope.
 4. The system of claim 1, further comprising a lightsource filter positioned in the light path of the light source thatsimultaneously transmits the excitation light and an amount of referencereflectance light not in a fluorescence detection wavelength band, theamount of reference reflectance light transmitted by the filter being afraction of the fluorescence excitation light, and wherein the lightsource filter blocks light from the light source at wavelengths in thefluorescence detection wavelength band from reaching the highsensitivity color image sensor to the extent that the light received bythe first group of pixel elements is substantially composed of lightresulting from tissue fluorescence and minimally composed of lightoriginating from the light source.
 5. The system of claim 4, the camerafurther comprising a blocking filter selectively positioned in front ofthe high sensitivity color image sensor for passing the fluorescencelight and the reflectance light and for blocking light the excitationlight.
 6. The system of claim 5, wherein the blocking filter blocksreflected excitation light, and transmits fluorescence and referencelight to the extent that the light received by the high sensitivitycolor image sensor is substantially composed of light resulting fromtissue fluorescence and reflected reference light and minimally composedof excitation light.
 7. The system of claim 5 wherein the fluorescencelight, transmitted by the blocking filter is green light.
 8. The systemof claim 7, wherein the reference reflectance light, not in the detectedfluorescence band, transmitted by the light source filter and theblocking filter is red light.
 9. The system of claim 7, wherein thereference reflectance light, not in the detected fluorescence band,transmitted by the light source filter and the blocking filter isnear-infrared light.
 10. The system of claim 5 wherein the fluorescencelight, transmitted by the blocking filter is red light.
 11. The systemof claim 10, wherein the reference reflectance light, not in thedetected fluorescence band, transmitted by the light source filter andthe blocking filter is green light.
 12. The system of claim 10, whereinthe reference reflectance light, not in the detected fluorescence band,transmitted by the light source filter and the blocking filter isnear-infrared light.
 13. The system of claim 1, further comprising alight source filter positioned in the light path of the light sourcethat transmits the fluorescence excitation light and blocks light atwavelengths in the fluorescence detection wavelength bands from reachingthe imaging areas of the high sensitivity color image sensor to theextent that the light received by the sensor is substantially composedof light resulting from tissue fluorescence and minimally composed oflight originating from the light source.
 14. The system of claim 1,further comprising a fluorescence/reflectance reference that producesknown levels of fluorescence and reflectance light upon illuminationwith light from the multimode light source, wherein the processor isprogrammed to adjust the gain of the sensor in response to the levels offluorescence and reflectance light produced.
 15. The system of claim 1,wherein the processor includes: a memory device that (i) stores asequence of instructions that cause the processor to adjust theintensity of the fluorescence or reference image signal on a pixel bypixel basis as a function of an analysis of the signals received fromthe first group of the pixel elements and the second group of pixelelements, and (ii) encodes the adjusted image signals received from thesensors as a video signal; and a video monitor for simultaneouslysuperimposing the separate video images.
 16. The system of claim 15,wherein the analysis of the image signals that the processor/controllercarries out utilizes a ratio of an intensity of the referencereflectance light and the fluorescence light received from the tissue.17. The system of claim 15, wherein the analysis of the image signalsthat the processor/controller carries out utilizes an intensity ofpixels that neighbor the one or more pixels and adjusts the processedimage signals based on the neighboring pixel intensities.
 18. The systemof claim 14, wherein the fluorescence/reflectance reference comprises adye and an amount of scatters in a solid.